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Claim of Priority 

5 This patent application claims the benefit of U.S. Provisional Application No. 

60/149 J96 filed August 19, 1999, which is hereby incorporated herein by reference in its 
entirety. 

Background of the Invention 

Field of the Invention 

0 The present invention relates to imaging systems, and in particular, to methods 

and systems for color interpolation. 
Description of the Related Art 

Conventional integrated circuit imaging devices include an array of fight 
detecting elements or pixels which are interconnected to generate a signal representation 

5 of an image illuminating the device. Two common examples of conventional integrated 

circuit imaging devices are a charge coupled device (CCD) and a complementary metal 
oxide semiconductor (CMOS) image sensing device. Conventional imaging devices use 
one or more light detecting elements and charge storage elements. In order to produce a 
color image, the imaging devices separate the light into various color components by 

;0 filtering the fight before the light strikes the light detecting elements. The array of fight 
detecting elements is often deposited with a filter layer such that neighboring pixels 
may have different color filters and organized in a particular pattern. 

Because each pixel is only capable of detecting a single color, conventional 
imaging devices require a process by which all of the color components are 

15 reconstructed for each pixel in order to maintain the original unfiltered array resolution. 

To reconstruct the color components, conventional imaging devices use a process of 
color interpolation that is performed after an analog signal associated with each pixel 
has been digitized. The conventional process of color interpolation performed after an 
analog signal associated with each pixel has been digitized requires conversion firom 

SO analog to digital (A/D) and may require extensive computations in order to achieve a 

high quality color presentation of the image. The A/D conversion and extensive 
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computations may require hardware, such as analog-to-digital (A/D) converters, 
memory, processors and software. The hardware and software may add to the 
complexity, size and expense of the imaging device and reduce the speed of the imaging 
process. 

5 Conventional imaging devices also require color compensation for differences 

in the response of the various color filters and for variations within the integrated circuit 
sensor array, such as process, materials, temperature or manufacturing. For example, 
when the primary color scheme is used, the response of an element that absorbs red 
light may be different than an element that absorbs blue hght even when illuminated by 

10 hght of equal red and blue luminosity levels. When exposed to a flat hght image 

having equal intensity and chromatisity levels throughout, typical CMOS or CCD 
arrays may generate analog signals having significant magnitude variations for the 
different color components. Accordingly, if the analog signals are used to reproduce 
the original image, the reproduced image colors will not match the original colors. 

15 To overcome this problem, conventional imaging devices employ a process of 

color correction that is performed after the analog signals for each pixel have been 
digitized. One drawback of the conventional color correction process is a loss in color 
dynamic range that results fi-om under-utihzation of the A/D converter for some of the 
color components. Another drawback is increased computations that translates into 

20 additional hardware, size, expense and/or reduced speed. 

The effect of loss in color dynamic range is particularly noticeable in the low 
hght areas of an image that contains low as well as high hght regions. The human eye 
is sensitive to minute changes in hue and saturation levels. A reduction in color 
component dynamic range may result in less vivid, plain, or flat images. Attempts to 

25 correct this via hue or saturation enhancement filters may cause color distortion rather 

than color restoration. 

Another color compensation that integrated circuit color imaging devices 
require is for different illumination temperatures. The hue of a color component 
changes with respect to the ambient illumination. Thus, a white object under sunhght 

30 conditions is perceived by the imaging device as white, but under fluorescent light 
conditions is perceived as light green. 
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To overcome this problem, conventional integrated circuit color imaging 
devices employ a process of white balance that is performed after the analog signals for 
each pixel have been digitized. Typically, each of the three processed color 
components, red, green or blue, is a convolution of the three colors. A drawback of the 

5 conventional white balance process is increased computations that translates into 

additional hardware and/or reduced speed. 

The collection of signals read from the pixels represents the image viewed by 
the array. Each pixel represents a sample of the image and hence is a data value in the 
two-dimension image produced by the imaging system. Defect pixels, referred to as 

10 'bad pixels,' do not contain a correct value and appear as artifacts. The bad pixels can 

reduce the image quality significantly. A bad pixel is caused by array defect and 
produces an output signal that significantly deviates from the mean output level of 
adjacent pixels when the exposure level of all pixels is substantially unified. Pixels that 
are significantly brighter than adjacent pixels in a unified dark frame are referred to as 

15 'hot pixels,' while pixels that are significantly darker than adjacent pixels in a unified 

bright frame are referred to as 'dead pixels.' 

The defect pixels are typically distributed in a random manner. However, a bad 
column (i.e. - a complete column is defective), or a blemish (i.e. - a cluster of 
neighboring pixels is defective) may occur and are typically discarded by the 

20 manufacturer. The manufacturer releases sensor arrays that contain random defect 
pixels in an amount that does not exceed a given limit, and the bad pixels are typically 
corrected. Both CCD and CMOS integrated circuit color imaging devices employ a 
process of bad pixel detection and correction. Conventionally, the detection step is 
performed off-line by the manufacturer. A bad pixel list is stored in the device. The 

25 correction step is typically performed after the analog signal for each pixel has been 

digitized. 

Summary of the Livention 
The present invention is directed to imaging methods and systems for flexibly 
30 addressing and processing imaging pixel sensor elements. The novel architecture of the 

present invention allows for a highly integrated, low cost imager with high speed 
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performance and good image quality. For example, the imaging system provides on- 
the-fly color interpolation, color compensation (also called color correction, color 
maximization or white balance) and/or fixed pattem noise reduction. The hardware 
and/or software related to on-the-fly color interpolation, color compensation and/or 

5 fixed pattem noise reduction may be provided on-chip. 

One embodiment of an imaging system in accordance v^th the present invention 
comprises an array of pixel sensor cells arranged in rows and columns, a plurality of 
detection circuits, an array controller, a control circuit that allows for various 
programmable modes of pixel readout, and a programmable amplification stage that 

10 may be adjusted per pixel or per group of pixels. In accordance with one embodiment, 
the imaging system further includes a color filter layer deposited on the sensor array. 
The color filter may comprise, for example. Red, Green, and Blue filters organized in 
the Bayer pattem scheme. In one embodiment^ the system fiirther contains a circuit for 
performing one or more averaging operations. 

15 One aspect of the present invention relates to an on-the-fly color interpolation 

apparatus and process that comprises a color reconstruction procedure performed during 
the sensor readout stage. The on-the-fly color interpolation process is capable of 
reading two or more consecutive rows and two or more consecutive columns 
simultaneously, summing some of the signals, and independently ampUfying the signals 

20 with an optionally programmable gain ampUfier. The on-the-fly color interpolation 
process is advantageously performed in an analog domain before an analog-to-digital 
conversion. 

The on-the-fly color interpolation process of the present invention provides high 
speed of operation and reduced computational complexity with good color image 

25 quaUty. The on-the-fly color interpolation process is advantageously suited for many 

imaging applications today that do not require high color precision or high image 
quality, as well as those which do. For example, some high quality, digital cameras 
have modes of operation such as a 'preview' mode where image quality is less important 
than speed and implementation complexity. 

30 One embodiment of the on-the-fly color interpolation process provides at least 

two modes of operation: a fiill resolution mode and a sub-sampling mode. Full 
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resolution mode is preferably used when high image quality is desired. In full 
resolution mode, each Red-Green-Blue (RGB) triplet needed for a color pixel 
representation is produced from a group of 2x2 pixels within the Bayer pattern. The 
four pixels are read out simultaneously, the two diagonally neighboring Green pixels 
are summed together, and the gain associated with the Green pixels is reduced in half. 
The resulting Green component is sent out with the diagonally neighboring Red and 
Blue pixels to produce one RGB triplet. Each subsequent RGB triplet shares one of the 
Green pixels and either the Red or Blue pixel of the preceding RGB triplet. 

The sub-sampling mode may be used when a lower resolution image is desired, 
such as when a user previews an image. Sub-sampling is achieved via skipping pixels 
along the horizontal and/or the vertical axis of the pixel array. For a sub-sampling ratio 
of l:j vertically and l:k horizontally, where j, k are even, each RGB triplet that is 
needed for a color pixel representation is produced from a group of j by k pixels with 
the Bayer pattern. All Green pixels in the j*k neighborhood are averaged to produce 
the green component. Similarly, all of the Red pixels in the same neighborhood are 
averaged to produce the Red component, and all Blue pixels in the neighborhood are 
averaged to produce the Blue component. The three components are then sent out as 
the RGB triplet representing the j by k group of pixels. The sub-sampling process can 
be achieved by reading the j rows simultaneously. 

Both the Ml resolution and the sub-sampling modes of operation can be further 
combined with a window readout mode where only a sub-region of the whole sensor 
array is read. 

Another aspect of the present invention relates to an apparatus and process for 
color compensation or maximization. The color compensation process compensates for 
differences in the response of various color filters and variations within the integrated 
circuit sensor array, such as process, materials, temperature or manufacturing. The 
color compensation process of the present invention improves color dynamic range, 
decreases computations and hardware and improves speed of operation. The color 
compensation process of the present invention is advantageously performed in the 
analog domain before an analog-to-digital conversion. The color compensation process 



of the present invention also improves white balance which is used to compensate 
colors for different illumination temperatures. 

In one embodiment, the imaging system with color compensation includes a 
control circuit that provides four output paths that can be ampUfied separately via one or 
more stages of programmable gain amphfiers and/or summing ampUfiers. The control 
circuit allows four independent readouts for even rows, odd rows, even columns and 
odd columns, thus providing separate and/or simultaneous output paths for Red, Green 
and Blue pixels. 

For example, one embodiment of the imaging system employs a readout control 
that provides outputs for the Red pixel, the Blue pixel, and the two Green pixels. Thus, 
the imaging system allows further gain compensation for a Green pixel that resides in an 
even column compared to a Green pixel that resides in an odd column. The four signals 
are then amplified via four corresponding programmable gain ampUfiers, The readout 
control logic ensures that the row and column switches are closed in the appropriate 
sequence. 

The present invention does not require the integrated color imaging system to 
employ a simultaneous readout of the n-by-n pixel block. A 'pipeline* approach may be 
utihzed instead of a parallel readout. The pipeline approach uses one or more analog 
line storage xmits, e.g., capacitors. For example, in one embodiment using the Bayer 
color pattern, two line storage units are used. The first of two consecutive lines that is 
readout from the array is stored in the first line storage unit. The second line is 
averaged with the stored line to produce the RGB triplets, while a "first" line of the next 
two consecutive lines is readout and stored in the second line storage unit and so on. 
Thus, the two line storage units are used in a 'ping pong' fashion. 

The imaging system does not restrict the type of transfer function that is 
implemented in the programmable gain amphfiers. Each of the four amplifiers can 
implement a different transfer function such as log or an exponent where the power 
value is progranmiable. Thus, each color can be optimized independently for maximum 
dynamic range. 

In one embodiment of the hnaging system with color compensation, the imaging 
system further employs several gain stages for color convolutions associated with white 
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balance. The imaging system employs a readout control that provides three outputs for 
the Red pixel, the Blue pixel, and either the even Green pixel or the odd Green pixel via 
a multiplexor. The three output signals are then amplified via nine programmable gain 
amphfiers and summed via three summing ampHfiers accordingly. The readout control 
logic ensures that the row and column switches are closed in the appropriate sequence. 

Another aspect of the invention relates to a fixed pattern noise reduction 
apparatus and process. The fixed pattern noise reduction process reduces noise related 
to pixel-to-pixel variation. This variation is primarily due to dark current leakage, 
which may be integrated together with the signal and hence contaminate the signal. The 
dark current leakage may be due to thermal generation in the neutral bulk material, in 
the depletion region and due to surface states. The dark current level may vary between 
pixels and may be particularly noticeable between columms due to column buffers. 

The fixed pattern noise reduction process of the present invention allows 
increased dynamic range (high image quahty), high speed of operation and reduced 
computational complexity. The fixed pattern noise reduction of the present invention is 
advantageously performed in the analog domain before an analog-to-digital conversion. 

In the imaging system of the present invention with fixed pattem noise 
reduction, the array of pixels comprises a group of exposed pixels and a group of dark 
pixels. In one embodiment, the dark pixels are deposited with an opaque mask layer 
and thus are not exposed to light. In one embodiment, the programmable readout 
control circuit has a programmable non-destructive readout mode. 

In one embodiment, the array includes a row of dark pixels. Each image pixel 
value is produced firom a combination of an exposed pixel and a dark pixel that resides 
in the same colimm. The two pixels are read simultaneously, and the dark value is 
subtracted fi-om the exposed value. In anotiier embodiment, the array includes several 
rows of dark pixels. 

In one embodiment, the imaging system produces a black and white image. In 
another embodiment, the imaging system fiirther comprises a color filter layer 
deposited on the exposed pixel sensor elements. In one embodiment, the color filter 
layer comprises Red, Green, and Blue filters organized in the Bayer pattem. In another 
embodiment, the color filter layer comprises Yellow, Cyan and Magenta filters. 
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In other embodiments, other color filter systems and/or other patterns or 
configurations may be used. In addition, other embodiments of the imaging system do 
not have pixels organized in a rectangular matrix. 

In one embodiment, at least a portion of the sensor cells are active. In another 
embodiment, at least a portion of the sensor cells are passive. 

The imaging system in accordance with the present invention may also include 
additional on-chip or off-chip amplification stages, analog-to-digital conversion units, 
memory units and various other signal processing blocks. In one embodiment, the 
imaging system fixrther comprises a micro-lenses layer. 

In one embodiment, the imaging system further contains a control circuit that 
allows for special pixel readout modes and a circuit for performing an averaging and/or 
a subtraction operation. 

In one embodiment, the imaging system employs an on-the-fly fixed pattern 
noise reduction process that subtracts dark current during the sensor readout stage. The 
on-the-fly fixed pattem noise reduction process is capable of reading two consecutive 
rows simultaneously by a readout shift register and a row readout control, re-reading a 
dark row together with an exposed row, subtracting a dark row value from an exposed 
row value with a summing amplifier and amplifying the difference with a 
programmable gain amphfier. 

In another embodiment, the imaging system employs a mode of on-the-fly fixed 
pattem noise reduction that subtracts a dark current average value during the sensor 
readout stage. The imaging system is capable of reading three consecutive rows 
simultaneously by a column shift register and a row readout control, re-reading the dark 
rows together with an exposed row, averaging the dark rows with a summing amphfier, 
subtracting the averaged dark row value from the exposed row value with a summing 
amphfier, and amphfying the difference with a programmable gain amphfier. The 
imaging system utilizes several dark rows for improved quality. Each image pixel 
value is produced from a combination of a current exposed pixel and a dark current 
value that is the average of two dark pixels residing in the same column. 

The present invention does not limit the number of dark rows that are averaged 
nor does it restrict the readout mode options. Furthermore, in other embodiments of the 
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present invention, the on-the-fly fixed pattern reduction can be performed with or 
without on-the-fly color interpolation, with or without sub-sampling, and can be further 
combined with a window readout mode where only a sub-region of the whole sensor 
array is utilized. 

5 Another aspect of the present invention relates to an integrated circuit imaging 

system that offers on-line bad pixel correction process that can be performed in the 
analog domain and thus provides high speed operation. 

Brief Description of the Drawings 
10 Figure 1 illustrates one embodiment of an imaging system coupled to a 

television. 

Figure 2 illustrates one embodiment of an imaging system coupled to a 
computer. 

Figure 3 illustrates a Bayer pattern color filter for the primary color system. 
1 5 Figure 4 illustrates one embodiment of a CMOS integrated circuit color imaging 

system that supports on-the-fly color interpolation. 

Figure 5 illustrates one embodiment of a readout sequence for the system of 
Figure 4 with on-the-fly color interpolation in a full resolution mode. 

Figure 6 illustrates one embodiment of the on-the-fly color interpolation process 
20 in a full resolution mode. 

Figure 7 illustrates one embodiment of a readout sequence for on-the-fly color 
interpolation in a sub-sampling mode. 

Figure 8 illustrates one embodiment of a CMOS integrated circuit color imaging 
system that supports on-the-fly color interpolation in a sub-sampling mode. 
25 Figure 9 illustrates one embodiment of the on-the-fly color interpolation process 

in a sub-sampling mode. 

Figure 10 illustrates a one embodiment of window mode. 

Figure 1 1 illustrates one embodiment of a readout control that supports on-the- 
fly color interpolation in a full resolution mode as shown in Figure 5. 
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Figure 12 illustrates one embodiment of a readout control that supports on-the- 
fly color interpolation in sub-sampling mode shown in Figure 8, as well as the full 
resolution mode shown in Figure 5. 

Figure 13 illustrates one embodiment of a CMOS integrated circuit color 
imaging system with color correction. 

Figure 14 provides one embodiment of a process for color correction. 

Figure 15 illustrates one embodiment of a CMOS integrated circuit color 
imaging system with white balance. 

Figures 16, 16A and 16B provide one embodiment of a process for white 
balance. 

Figure 17 presents embodiments of optional transfer function for the 
programmable gain ampHfiers. 

Figures 18, 18A and 18B provide an another embodiment of the process shown 
in Figure 16. 

Figure 19 illustrates another embodiment of a CMOS integrated circuit color 
imaging system with white balance. 

Figure 20 illustrates one embodiment of an imaging system that supports fixed 
pattern noise reduction. 

Figure 21 illustrates one embodiment of a fixed pattern noise reduction process. 

Figure 22 illustrates another embodiment of an imaging system that supports 
fixed pattem noise reduction. 

Figure 23 illustrates another embodiment of a fixed pattem noise reduction 
process. 

Figure 24 illustrates one embodiment of a CMOS integrated circuit imaging 
device implementation that accommodates on-the-fly bad pixel correction. 

Figure 25 illustrates a flowchart for the on-the-fly bad pixel correction of Figure 

24. 

Detailed Description of the Preferred Embodiments 
The present invention relates to a novel imaging system that provides flexible 
addressing and processing of imaging pixel sensor elements. The novel architecture of 
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the present invention allows for a highly integrated, low cost imager with high speed 
performance and good image quality. For example, the imaging system may provide 
on-the-fly color interpolation, color compensation (also called color correction, color 
maximization or white balance) and/or fixed pattern noise reduction. 

The exempUfying imaging systems described below with reference to Figures 1- 
23 use a CMOS integrated circuit, an array of pixels organized in a rectangle matrix, 
and a color filter with a primary color system (RGB) in a Bayer color pattern. The 
imaging systems of the present invention may be implemented with a charge coupled 
device (CCD) or other imaging technologies. Likewise, the imaging systems of the 
present invention may be implemented with another color system, such as the 
complimentary color system (Yellow, Cyan and Magenta) and/or another color pattern. 
In addition, the imaging system of the present invention may be implemented with the 
pixels organized in another pixel matrix or pixel topography. For example, the array 
does not have to be rectangular. 

The imaging system may also include either on-chip, as is possible with CMOS 
integrated circuit imaging devices, or off-chip, as is the case with CCD integrated 
circuit imaging devices, amplification stages, analog to digital conversion units, 
memory units and various other signal processing blocks. In addition, the system may 
further comprise a micro-lenses layer. 

In one embodiment, the color interpolation system, the color compensation 
system and/or the fixed pattern noise reduction system reside together with the sensor 
array in the same chip, such as in a CMOS integrated circuit color imaging device. In 
another embodiment, the color interpolation system, the color compensation system 
and/or the fixed pattern noise reduction system reside in a separate companion chip, 
such as in a CCD integrated circuit unaging device. 

Figures 1 and 2 illustrate exempHfying systems incorporating the novel imaging 
system 100, 102. The imaging systems 100, 102 will be described in greater detail 
below with reference to Figures 3-23. 

Figure 1 illustrates one embodiment of an imaging system 100 coupled to a 
television 1 16 via a coax cable 114. The exemphfying system 100 includes a lens 102, 
a sensor array 104 which may have color filters, a readout control 106, gain amplifiers 
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108 for each color, a NTSC encoder 110 including gamma correction and a power 
supply 112. In one embodiment, the imaging system 100 is a video camera. In other 
embodiments, the imaging system 100 may be implemented in security cameras, digital 
cameras, camcorders, video telephones and the like. 

Figure 2 illustrates one embodiment of an imaging system 122 coupled to a 
computer 126 via a USB cable 124. The exemphfymg system 122 includes a lens 102, 
a sensor array 104 which may have color filters, a readout control 106, gain amplifiers 
108 for each color, an analog to digital converter 1 18, a USB interface 120 and a power 
supply 112. In one embodiment, the imaging system 122 is a video camera. In other 
embodiments, the imaging system 122 may be implemented in security cameras, digital 
cameras, camcorders, video telephones and the like. 

Figure 3 illustrates a conventional Red, Green and Blue (primary color system) 
Bayer color pattern 130 for a color filter that is deposited on an array of pixel cells that 
detect Ught. The pattern core is a group of 2 by 2 pixels that contains 2 green 
components 134, 136, one red component 132 and one blue component 138. The 2 
green components 134, 136 are diagonal neighbors, and the red and the blue 
components 132, 138 are diagonal neighbors. Thus, the green resolution of the array is 
reduced by a ratio of 2:1 horizontally only, while the red and the blue resolution is 
reduced by a ratio of 2:1 horizontally and vertically. The pattern 130 exploits the fact 
that the human eye perceives intensity edges better than color edges and that the green 
component contains the highest amount of intensity information. 
On-The-Flv Color Interpolation 

As will now be described, an imaging system with a flexible pixel address 
scheme allows for on-the-fly interpolation of colors based on the outputs of two or 
more pixels. On-the-fly color interpolation relates to color reconstruction during the 
sensor readout stage. Color reconstruction relates to reconstructing desired color 
components for each pixel in order to maintain the original imfiltered array resolution 
and to compensate for the fact that each filtered pixel is only capable of detecting a 
single color. 

The on-the-fly color interpolation process of the present invention provides high 
speed of operation and reduced computational complexity with good color image 
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quality. The on-the-fly color interpolation process is advantageously suited for many 
imaging applications that do not require high color precision or high image quality, as 
well as those which do. For example, some high quaUty, digital cameras have modes of 
operation such as a 'preview' mode where image quality is less important than speed 
and implementation complexity. 

One embodiment of the on-the-fly color interpolation process provides at least 
two modes of operation: a full resolution mode and a sub-sampling mode. Full 
resolution mode is preferably used when high image quality is desired. In full 
resolution mode, each Red-Green-Blue (RGB) triplet needed for a color pixel 
representation is produced from a group of 2x2 pixels within the Bayer pattern. The 
four pixels are read out simultaneously, the two diagonally neighboring Green pixels 
are summed together, and the gain associated with the Green pixels is reduced in half. 
The resulting Green component is sent out with the diagonally neighboring Red and 
Blue pixels to produce one RGB triplet. Each subsequent RGB triplet shares one of the 
Green pixels and either the Red or Blue pixel of the preceding RGB triplet. 

Sub-sampling mode is preferably used when a lower resolution image is desired, 
such as a preview feature. Sub-sampling is achieved via skipping pixels along the 
horizontal and/or the vertical axis of the pixel array. For a sub-sampling ratio of l:j 
vertically and l:k horizontally, where j, k are even, each RGB triplet that is needed for a 
color pixel representation is produced firom a group of j by k pixels with the Bayer 
pattern. All Green pixels in the j*k neighborhood are averaged to produce the green 
component. Similarly, all of the Red pixels in the same neighborhood are averaged to 
produce the Red component, and all Blue pixels in the neighborhood are averaged to 
produce the Blue component. The three components are then sent out as the RGB 
triplet representing the j by k group of pixels. The sub-sampling process can be 
achieved by reading the j rows in parallel or in series. 

Both the full resolution and the sub-sampling modes of operation can be fiirther 
combined with a window readout mode (also called "window mode" or "windowing"). 
In window mode, a sub-region of the whole pixel sensor array is readout and processed. 
Thus, a window mode is essentially a cropping operation that produces a smaller area of 
interest. Window readout mode is faster and provides a higher frame rate. Window 
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mode may be used in digital cameras for exposure and focus calculations, for electronic 
zoom and more. Window mode is described in greater detail below with reference to 
Figure 10. 

Figure 4 illustrates one embodiment of a novel CMOS integrated circuit color 
5 imaging system 140 that supports on-the-fly color interpolation using analog signals. 

The system 140 uses the Bay color pattern illustrated in Figure 3. As illustrated in 
Figure 4, the system 140 includes a column readout control circuit 146, a line 
synchronization signal 142, a pixel clock 144, a first column readout line 148, a second 
column readout line 148', a third column readout line 148", a first switch 150, a first 
10 column buffer 152, a second switch 150', a second column buffer 152', an analog 

summing ampUfier 154, a programmable analog red gain amplifier 156, a red video out 
line 158, a red gain control line 160, a green video out line 162, a green gain control 
line 164, a programmable analog green gain amphfier 166, a blue video out line 168, a 
blue gain control line 170, a programmable analog blue gain amplifier 172, a red output 
15 path 174, a green output path 176, a blue output path 178, a row readout control circuit 
180, a firame synchronization line 182, a line synchronization line 184, a first row 
readout line 186, a second row readout line 186\ a first row buffer 188, a second row 
buffer 188', a first red pixel 190, a first green pixel 192, a second green pixel 194 and a 
first blue pixel 196. 

20 In the description herein, a "programmable" component refers to a component 

that responds to a command from an end-user of the imaging system or to a command 
issued by intemal firmware according to firmware stored in the imaging system. For 
example, if an end-user chooses a 'zoom' fimction or a 'preview' function on a video 
camera containing the imaging system of the present invention, the imaging system 

25 directs the programmable components to act in a predefined manner according to 

firmware stored in the imaging system. 

The imaging system of the present invention does not need the color gain 
ampUfiers 156, 166, 172 (Figure 4) to be implemented as a separate stage. In one 
embodiment, a plurality of color gain amplifiers are contained within the pixel circuitry 

30 of the sensor array. In another embodiment, a pluraUty of color gain amphfiers are 
contained within the column buffers 152, 152'. 



14 



As will now be described, the system 140 accommodates on-the-fly color 
interpolation via a programmable pixel readout mode. The exemplifying system 140 
implements an RGB Bayer color pattern on-the-fly color interpolation when windowing 
and sub-sampling are not active. The system 140 is not limited in its mode of operation 
5 and can support windowing and sub-sampling via the programmable readout control 

circuitry 146, 180, the summing amplifier(s) 154 and the programmable gain ampHfiers 
156, 166, 172. 

Figure 5 illustrates one embodiment of a pixel readout sequence for the system 
140 of Figure 4 with on-the-fly color interpolation in a full resolution mode. The 
10 sequence accommodates an RGB Bayer pattern color filter 130 (Figure 3) in a non- 
window mode and without sub-sampling. Each RGB triplet that is needed for a color 
pixel representation is produced from a group of 2x2 pixels that contains the Bayer 
pattem core. For example, while reading four pixels substantially simultaneously, the 
two diagonally neighboring Green pixels 192, 194 are averaged and sent out with the 
15 neighboring Red and Blue pixels 190, 196 to produce a RGB triplet. Three other 

subsequent RGB triplet shares one of the Green pixels 192, 194 and either the Red or 
the Blue pixels 190, 196 with the previous RGB triplet. For example, in Figure 5, four 
RGB triplets are formed from a 3x3 pixel block 200: 
{R(0,0), [(G(1,0) + G(0,l))/2], B(l,l)} 
20 {R(2,0), [(G(1,0) + G(2,l))/2], B(l,l)} 

{R(0,2), [(G(0,1) + G(l,2))/2], B(l,l)} 
{R(2,2), [(G(2,1) + G(1,2))/2],B(1,1)} 
where the first numeral in the parenthesis represents the row and the second numeral 
represents the column. 

25 Similar readout sequences can be devised in accordance with the present invention for 
color filter patterns other than Bayer and for color systems other than the primary one. 

Figure 6 illustrates one embodiment of the on-the-fly color interpolation process 
in a full resolution mode. In a process block 202, the illustrated system 140 may 
simultaneously read out a group of 2 by 2 pixels that contains the Bayer pattem core. 

30 In a process block 203, the system 140 may perform several acts substantially 
simultaneously (in parallel). In blocks 204, 210, the system 140 reads the 
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corresponding red and the blue components 190, 196 that reside in two consecutive 
colimms 148, 148" (Figure 4) and amplifies accordingly via the programmable red and 
blue gain amphfiers 156, 172. In blocks 206-208, the system 140 reads the two green 
components 192, 194 that reside in two consecutive columns 148, 148" and sums the 
5 values with the summing amplifier 154, The gain from the summing ampUfier 154 is 
adjusted accordingly via the programmable gain amplifier 166. The programmable 
readout control 146, 180 ensures that the appropriate switches are closed to allow the 
correct readout sequence as described in Figure 5. 

In Figure 6, the acts in block 203 (ampUfying and outputting a red value, 

10 ampUfying and outputting a green value and amplifying and outputting a blue value) 

are performed substantially simultaneously. In another embodiment, the acts of block 
203 are performed in a sequence, e.g. the system 140 amphfies and outputs a red value, 
then amphfies and outputs a green value, and then amphfies and outputs a blue value. 

In Figure 6, the system 140 advances to the next column in a block 212, new 

15 COLUMN = old COLUMN + 1. hi a block 214, the system 140 determines whether 

the column readout control 146 is exceeding the last Red pixel in a row. If yes, then the 
process proceeds to a block 216, and the row readout control 180 advances to the next 
row, new ROW = old ROW + 1. If not, then the process loops back to block 202. In a 
block 218, the system determines whether the row readout control 180 is exceeduig the 

20 last Red pixel in a frame. If yes, then the process stops in a stop block 220 and waits 
for further commands. If not, then the process proceeds to block 202, 

Figure 7 illustrates one embodiment of a readout sequence for on-the-fly color 
interpolation in a sub-samphng mode. In Figure 7, the sequence accommodates an 
RGB Bayer pattern color fiher 230 in a non-window mode with sub-sampling. In 

25 Figure 7, the sub-samphng ratio is 1:4 horizontally and 1:4 vertically. Each RGB 

triplet that is needed for a color pixel representation is produced from a group 232 of 
4x4 pixels that contains 4 Bayer pattern cores. The 8 green pixels are averaged to 
produce the green component, the 4 red pixels are averaged to produce the red 
component, and the 4 blue pixels are averaged to produce the blue component. 

30 Specifically, as illustrated in Figure 7, the red component comprises 

[(R(0,0) + R(2,0)+ R(0,2) + R(2,2))/4], 
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the green component comprises 

[(G(UO) + G(3,0) + G(04) + G(2,l) + G(l,2) + G(3,2) + G(0,3) + G(2,3))/8], 
and the blue component comprises 

[(B(14) + B(3,l) + B(l,3) + B(3,3))/4] 

5 Similar readout sequences can be devised for color filter patterns other than Bayer and 

for color systems other than the primary one. 

Figure 8 illustrates one embodiment of a CMOS integrated circuit color imaging 
system 240 that supports on-the-fly color interpolation in a programmable pixel readout 
mode, such as a sub-sampling mode. The structure of the system 240 is substantially 

10 similar to the system 140 of Figure 4, except that the system 240 also includes a red 

summing amphfier 242 and a blue summmg amplifier 244. The fimction of these 
additional amplifiers is described below. Figure 8 presents an implementation for an 
RGB Bayer pattem on-the-fly color interpolation where windowing is not active and 
sub-sampling is active. However, the system 240 is not limited to a Bayer pattem 

15 and/or the primary color scheme. Nor is the system 240 limited in its mode of 

operation. The system 240 can be realized to support both window and sub-sampling 
via the progranmiable readout control circuitry 146, 180, the summing amphfier(s) 242, 
154, 244, and the programmable gain amplifiers 156, 166, 172. 

Figure 9 illustrates one embodiment of the on-the-fly color interpolation process 

20 in a sub-sampling mode, as described with reference to Figures 7 and 8. Figure 9 
illustrates sub-sampling a RGB Bayer color filter 230 (Figure 7) using a ratio of 4:1 
horizontally and 4:1 vertically. In a block 250, a group 232 (Figure 7) of 4 by 4 pixels 
that contains Bayer pattem cores is read out simultaneously. 

In a process block 253, the system 240 may perform several acts substantially 

25 simultaneously (in parallel). In blocks 252, 254, the four red pixels shown in sub- 

sample 232 of Figure 7 are summed and amphfied by the red ampUfiers 242, 156 of 
Figure 8. In blocks 256, 258, the 8 green pixels that reside in four consecutive columns 
(Figure 7) are summed via the green summing amplifier 154 and amplified by the green 
ampUfier 166 (Figure 8). In blocks 260, 262, the four blue pixels shown in sub-sample 

30 232 of Figure 7 are summed and amplified by the blue amphfiers 244, 172 of Figure 8. 

The programmable readout control 146, 180 of Figure 8 ensures that the appropriate 
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switches are closed to allow the correct readout sequence as described in Figures 7 and 
9. 

In Figure 9, the acts in block 253 are performed substantially simultaneously. 
In another embodiment, the acts of block 253 are performed in a sequence, e.g. the 
5 system 240 sums, amplifies and outputs a red value, then sums, amphfies and outputs a 
green value, and then sums, amphfies and outputs a blue value. 

In Figure 9, the system 240 advances to the next 4x4 block horizontally in a 
block 264, new COLUMN = old COLUMN + 4. In a block 266, the system 240 
determines whether the column readout control 146 is exceeding the last 4x4 block in a 

10 row. If yes, then the process proceeds to a block 268, and the system 240 advances 

vertically to the next 4x4 block, new ROW = old ROW + 4. If not, then the process 
loops back to block 250. In a block 270, the system 240 determines whether the row 
readout control 180 is exceeding the last 4x4 block in a frame. If yes, then the process 
stops in a stop block 272 and waits for further commands. If not, then the process 

1 5 proceeds to block 250. 

Figure 10 illustrates a one embodiment of window readout mode. For 
simplicity, sub-sampling and/or on-the-fly color interpolation are not included in the 
embodiment shown in Figure 10. In window mode, a sub-region 280 of the pixel array 
exposed area 130 is readout. In the embodiment shown in Figure 10, the pixel array 

20 exposed area size is n by m, the window size is chosen to be n - 4 by m - 6, and the 
window origin is chosen to be (4,4). Thus, the sub-region 280 comprises pixels in a 
rectangle defined by four pixels R(4,4), G(m-3,4), G(4,n) and B(m-3,n) at the four 
comers. 

A window mode is essentially a cropping operation that produces a smaller area 
25 of interest. The readout is faster and provides a higher frame rate than the system 240 

of Figure 8. The system 240 of Figure 8 may be modified to support window readout 
mode. Window mode may be used in digital cameras for exposure and focus 
calculations, for electronic zoom and more. 

Figure 11 illustrates one embodiment of a column readout control 146 (Figure 
30 4) that supports on-the-fly color interpolation in a full resolution mode as illustrated in 
Figure 5. The readout control of the system 146 advantageously allows the 
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simultaneous readout of a 2x2 pixel block. The readout control comprises a column 
readout control 146 that provides simultaneous readout of two columns and a row 
readout control 180 that provides a simultaneous readout of two rows. 

The column readout control 146 includes a shift register that is responsible for 
5 closing the switches of the column buffers two at a time. The pixel clock 144 clocks 
the shift register 146. The Une sync signal 142 is feeding the shift register 146 and is 
multiphed by two to allow signal width of two consecutive bits. Thus, the shift register 
146 closes two consecutive column buffer switches coupled to lines 148, 148' 
simultaneously. 

10 The row readout control 180 (Figure 4) is substantially similar except the shift 

register clock of the row readout control 180 utiHzes a line sync 184 instead of the pixel 
clock, and the shift register input data utihzes a frame sync 182 instead of the line sync 
signal. 

Figure 12 illustrates one embodiment of a novel readout control that supports 
15 on-the-fly color interpolation in sub-sampling mode shown in Figure 8, as well as the 

fill! resolution mode shown in Figure 5. Figure 12 demonstrates the column readout 
control 146. The sub-sampling modes chosen for the example are: 1:2, 1:4, 1:8 and 
1:16. The readout control of Figure 12 also supports overlap such as the case in the fiiU 
resolution mode (Figure 5). The shift register that is responsible for closing the column 
20 buffer switches is divided into 16-bit registers 318, 318\ 318" that are clocked by the 
pixel clock 324. 

In Figure 12, each register 318 is loaded with a bit pattern that is produced by 
the pattern generator 290. In a fiiU resolution mode, the pattem generator 290 shifts a 
pattern 292 (provided by the initial pattem signal line) left by 1 during each cycle and 

25 causes an overlap. For a sub-sampling ratio of 1 :4, "1 1 11" is the pattem 292 provided 
on an initial pattem line and loaded since four colunms are read simultaneously. In 
each cycle, the pattem is shifted left and selected by the multiplexors 304, 308. For 
example, a shift amount of 4 is used for sub-sampling 1:4 without overlap. The down 
counter 314 is responsible for counting pixel clocks for each 16-bit register 318, e.g. for 

30 sub-sampUng of 1:4 without overlap, the counter 314 counts 4 times from the line sync 

322. Then the counter 314 is reloaded by the register 312, and the next 16-bit register 
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318' is enabled by the latch 316 and the initial pattern is reloaded by the 16-bit register 
308. 

The row readout control 180 is similar to the column readout control 146 shown 
in Figure 12 except the clock 324 utilizes a line sync 184 instead of the pixel clock, and 

5 the initializing signal 322 utilizes a frame sync 182 instead of the line sync signal. 
Color Compensation 

As previously discussed, another aspect of the invention relates to color 
compensation (correction or maximization). The color compensation process 
compensates for differences in the response of various color filters and variations 

10 within the integrated circuit sensor array, such as process, materials, temperature or 

manufacturing. The color compensation process of the present invention improves 
color dynamic range, decreases computations and hardware and improves speed of 
operation. The color compensation process of the present invention is advantageously 
performed in the analog domain before an analog-to-digital conversion. 

15 Figure 13 illustrates one embodiment of a CMOS integrated circuit color 

imaging system 330 of the present invention with analog color correction. The system 
330 provides a programmable pixel readout mode that allows four independent readouts 
and four programmable gain amplifiers 156, 172, 342, 352 for ampHfying the four 
output paths (red, blue, even green, odd green) separately. Figure 13 presents an 

20 implementation for an RGB Bayer pattern. However, the system 330 is not limited to a 
Bayer pattem and/or the primary color scheme. In Figure 13, the system 330 employs 
readout control circuitry 146, 334 with separate paths for even and odd rows and 
columns 174, 178, 348, 350, and programmable gain amplifiers 156, 172, 342, 352. 
Thus, the system 330 provides four separate, simultaneous output paths with potentially 

25 different amplified gains. 

The system 330 is not restricted to a particular type of transfer fimction with 
respect to the programmable gain amplifiers 156, 172, 342, 352. In one embodiment, 
each of the four programmable gain amplifiers 156, 172, 342, 352 implement different 
transfer functions, such as a log or an exponent fimction where the power value is 

30 programmable, as shown in Figure 17. Thus, the different transfer fimctions 

advantageously allow different levels of color compensation for different colors. In one 
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embodiment, the transfer functions may be modified in real-time according to the level 
of compensation needed to accommodate temperature, differences in the response of 
various color filters and variations within the integrated circuit sensor array, such as 
process, materials or manufacturing. In another embodiment, the transfer functions may 

5 be pre-set at the manufacturer to accommodate variations within the integrated circuit 

sensor array, such as process, materials or manufacturing. 

Figure 14 provides one embodiment for the analog color correction process 
performed by the system 330 of Figure 13. In a block 360, the system 330 (Figure 13) 
reads a pixel value. In a block 363, the system 330 may perform several acts 

10 simuhaneously (in parallel). Blocks 362, 368 and 372 determine the color of the pixel 

value. In a block 364, the red pixel 190 that resides in an even column and even row is 
amplified via the red programmable amphfier 156. In a block 366, the green pixel 192 
that resides in an even column and odd row is ampUfied via the even green 
programmable amplifier 342. In a block 370, the green pixel 194 that resides in an odd 

15 column and even row 194 is amplified via the odd green amphfier 352. In a block 374, 
the blue pixel 196 that resides in an odd column and odd row is amplified via the blue 
programmable amplifier 172. The acts in block 363 are performed substantially 
simultaneously. In another embodiment, the acts of block 363 are performed in a 
sequence. 

20 In a block 376, the system 330 advances to the next column, new COLUMN = 

old COLUMN +1, In a block 378, the system 330 determines whether the system 
readout is exceeding the last pixel in a row. If not, the system 330 returns to process 
block 360. If yes, the system 330 advances to the next row, new ROW old ROW + 1, 
in a block 380. In a block 382, the system 330 determines whether the system readout 

25 is exceeding the last row in a frame. If not, the system 330 returns to process block 360. 

If yes, the system 330 stops in a stop block 384 and waits for further commands. 

In addition to the significant innovations described above, one embodiment of 
the color compensation process of the present invention also improves white balancing, 
which is used to compensate colors for different illumination temperatures. 

30 Figure 15 illustrates one embodiment of a CMOS integrated circuit color 

imaging system 390 that accommodates analog white balance with programmable pixel 
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readout modes, multiplexor(s) 418, programmable gain amplifiers 398-414, and 
summing amplifiers 392, 394 and 396. Figure 15 presents an implementation for an 
RGB Bayer pattem. However, the system 390 is not limited to a Bayer pattern and/or 
the primary color scheme. In Figure 15, the system 390 employs readout control 
5 circuitry 146, 180 with separate paths for even and odd rows 186 and columns 148, a 

multiplexor 418 for selecting the appropriate green path, programmable gain amplifiers 
398-414, and summing amplifiers 392, 394 and 396. The multiple amplifier gain stages 
provide convolutions associated with white balance. 

Figures 16, 16A and 16B (hereinafter referred as "Figure 16") provide one 

10 embodiment of an analog white balance process performed by the system 390 of Figure 

15. In a block 420, the system 390 reads a 2x2 pixel block. In a process block 423, the 
system 390 may perform several acts substantially simultaneously (in parallel). In 
blocks 422 and 430, the system 390 determines the colors of the pixels. In blocks 424- 
428, the red pixel 190 is amphfied by 3 red programmable ampUfiers 398-402 (Figure 

15 15). In blocks 440, 447 and 444, the blue pixel 196 is amphfied by 3 blue 
programmable ampUfiers 410-414. In a block 432, the green pixel 192 that resides in an 
even column and odd row is multiplexed by the multiplexor 418 with the green pixel 
194 that resides in an odd column and even row to produce one green value that is 
amphfied by 3 green ampUfiers 404-408. In blocks 446, 448 and 450, each of the 

20 amplified red components is summed with the corresponding amplified green and 

amplified blue components via 3 summing ampUfiers 392, 394 and 396 (Figure 15), and 
thus producing new red, green and blue components that are results of RGB 
convolutions. 

In a block 452, the system 390 advances to the next column, new COLUMN = 
25 old COLUMN +1. In a block 454, the system 390 determines whether the system 

readout is exceeding the last pixel in a row. If not, the system 390 returns to process 
block 420. If yes, the system 390 advances to the next row, new ROW = old ROW + 1, 
in a block 456. In a block 458, the system 390 determines whether the system readout 
is exceeding the last row in a firame. If not, the system 390 returns to process block 420. 
30 If yes, the system 390 stops in a stop block 460 and waits for further commands. 
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The acts in block 423 are performed substantially simultaneously. For example, 
the decision blocks 422 and 430 may be performed in parallel by reading two or more 
pixels simultaneously. As another example, blocks 424, 426, 428, may be performed in 
parallel by themselves or with blocks 440, 447, 444 and/or blocks 434, 436, 438. In 
5 other embodiments, the blocks may be performed in a sequence. The system 390 may 

further comprise storage circuits to store the values of the ampHfied red, green and blue 
components before they are summed in blocks 446, 448 and 450. 

Figure 17 presents embodiments of optional transfer functions 470, 472, 474, 
such as an exponent transfer function, for the programmable gain amplifiers 398-414 of 

10 Figure 15. Each amplifier can implement a different transfer function and thus optimize 
each color for maximum dynamic range. 

Figures 18, 18A and 18B (hereinafter referred as "Figure 18") provide another 
embodiment of the process shown in Figure 16. The process of Figure 18 provides 
white balance and on-the-fly color interpolation. The process may be performed by the 

15 system 390 of Figure 19. The process of Figure 18 is similar to the process shown in 

Figure 16, except the two green pixels are averaged by a summing amplifier 482 
(Figure 19) in a block 432 of Figure 18. In Figure 18, like the process in Figure 16, 
some of the process blocks are performed in parallel. In other embodiments, the blocks 
may be performed in a sequence, 

20 Figure 19 is another embodiment of a CMOS integrated circuit color imaging 

system 390 of the present invention with white balance. The system 390 of Figure 19 is 
similar to the system 390 of Figure 15, except the system 390 of Figure 19 comprises a 
summing amplifier 482 instead of a multiplexor 418 in Figure 15. The system 390 of 
Figure 19 provides white balance and on-the-fly color interpolation. 

25 Fixed Pattern Noise Reduction 

Another aspect of the invention relates to a fixed pattern noise reduction 
apparatus and process. The fixed pattem noise reduction process reduces noise related 
to pixel-to-pixel variation. This variation is primarily due to dark current leakage, 
which may be integrated together with the signal and hence contaminate the signal. The 

30 dark current leakage may be due to thermal generation in the neutral bulk material, in 
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the depletion region and due to surface states. The dark current level may vary between 
pixels and may be particularly noticeable between columns due to column buffers. 

The fixed pattern noise reduction process of the present invention allows 
increased dynamic range (high image quality), high speed of operation and reduced 
5 computational complexity. The fixed pattern noise reduction of the present invention is 
advantageously performed in the analog domain before an analog-to-digital conversion. 

Figure 20 illustrates one embodiment of an imaging system that supports fixed 
pattern noise reduction. Specifically, Figure 20 illustrates one embodiment of a CMOS 
integrated circuit color imaging system 500 that supports on-the-fly fixed pattern noise 

10 reduction. In the illustrated embodiment, additional circuitry is not needed for post- 
processing in order to provide the fixed pattern noise reduction. The system 500 
comprises a column readout shift register 146, a row readout shift register 180, a dark 
row readout control or path 492, a row of dark pixel sensor elements (dark row) 490, a 
first row of exposed pixel sensor elements 186, a second row of exposed pixel sensor 

15 elements 186', a summing amplifier 496, a programmable gain amphfier 498 and a 
plurality of switches 508. 

In one embodiment, the dark pixels are deposited with an opaque mask layer 
and thus are not exposed to light. A row of pixels with dark pixels may be referred to 
as a "dark row." The system 500 presents an implementation for a red/greenl3lue 

20 (RGB)(primary color scheme) in a Bayer pattern with on-the-fly fixed noise reduction 

when windowing and sub-sampling are not active. 

The system 500, however, is not limited to a Bayer pattem and/or the primary 
color scheme. Other embodiments may use other color schemes and/or other color 
patterns. Nor is the system 500 limited in its mode of operation. For example, the 

25 system 100 of Figure 1 may be modified to support windowing and/or sub-sampling. 

In Figure 20, two pixels in the same column are read out simultaneously in 
response to signals fi-om the column shift register 146, the row shift register 180 and the 
dark row readout control 492. One pixel is read firom the dark row 490 and another 
pixel is read from the first exposed row 186. The dark pixel value is then subtracted 

30 fi*om the exposed pixel value by the summing amplifier 496, thereby providing fixed 

pattem noise reduction. Additional gain is applied to the new value by the 
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programmable gain amplifier 498. The dark row 490 may be re-used for each exposed 
row 186, 186'. The programmable readout control components 146, 180 ensure that the 
appropriate switches 508 are closed to allow the correct readout sequence. 

Figure 21 illustrates one embodiment of a fixed pattem noise reduction process 
5 that may be performed by the system 500 of Figure 20, Specifically, Figure 21 

illustrates one embodiment of an on-the-fly fixed pattem noise reduction process 
utilizing a single dark row 190. In Figure 21, in a process block 520, the system 500 
reads two pixels in a current column: a dark pixel in the dark row 490 and an exposed 
pixel in an exposed row 186. In a block 522, the summing amplifier 496 subtracts a 

10 value associated with the dark pixel from a value associated with the exposed pixel. In 
a block 524, the system 500 clips the difference found in block 522 to zero if the 
difference is negative. In a block 526, the system 500 advances to the next column of 
the system 500 in Figure 20, new COLUMN - old COLUMN + 1 . 

In a decision block 528, the system 500 determines whether the column shift 

15 register 146 has reached the last pixel in a row 186. If the column shift register 146 has 

not reached the last pixel in a row 186, then the system 500 retums to process block 520 
to process the next column. If the column shift register 146 has reached the last pixel in 
a row 186, then the system 500 advances to the next row 186' in a process block 530, 
new ROW = old ROW +1. In a decision block 532, the system 500 determines 

20 whether the row shift register 180 has reached the last row in a frame of pixel sensor 

elements. If the row shift register 180 has not reached the last row in a fi-ame of sensor 
elements, then the system 500 retums to process block 520 to process the first column 
in the next row 186'. If the row shift register 180 has reached the last row in the fi"ame, 
then the system 500 may wait for a further command in an end block 534. 

25 Figure 22 illustrates another embodiment of an imaging system 540 that 

supports fixed pattem noise reduction. Specifically, Figure 22 illustrates one 
embodiment of a CMOS integrated circuit color imaging system 540 that supports on- 
the-fly fixed pattem noise reduction with two dark rows. The imaging system 540 uses 
more than one dark rows for improved quality. Each image pixel value is preferably 

30 produced fi*om a combination of a current exposed pixel and a dark current value that is 
the average of two dark pixels residing in the same column. 
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In Figure 22, the system 540 comprises a column readout shift register 146, a 
row readout shift register 180, a dark row readout control or path 492, a first row of 
dark pixel sensor elements (first dark row) 490, a second row of dark pixel sensor 
elements (second dark row) 544, a first row of exposed pixel sensor elements 186, a 
5 second row of exposed pixel sensor elements 186\ a first summing amplifier 542, a 
second summing amplifier 496, a programmable gain amplifier 498 and a plurality of 
switches 508. The system 540 presents an implementation for a red/green/blue 
(RGB)(primary color scheme) in a Bayer pattern with on-the-fly fixed noise reduction 
when windowing and sub-samphng are not active. 

10 In another embodiment, the dark rows 490, 544 and/or individual dark pixels are 

distributed over the imaging array. 

The system 540 is not limited to a Bayer pattern and/or the primary color 
scheme. Other embodiments may use other color schemes and/or other color patterns. 
Nor is the system 540 limited in its mode of operation. For example, the system 540 of 

1 5 Figure 22 may be modified to support window and sub-sampling. 

In Figure 22, three pixels in the same column are read out simultaneously by the 
colimm shift register 146, the row shift register 180 and the dark rows readout path 492. 
Two pixels are read from the two dark rows 490, 544 and a third pixel is read from the 
first exposed row 186. The two dark pixels are then averaged by the first summing 

20 amplifier 542 to produce an average dark value. The average dark value is then 
subtracted from the exposed pixel by the second summing amplifier 496. Additional 
gain is apphed to the new value by the programmable gain amplifier 498. The dark row 
490 is re-used for each exposed row 186. The programmable readout control 
components 146, 180 ensure that the appropriate switches 508 are closed to allow the 

25 correct readout sequence. 

In Figure 22, the system 540 has two dark rows 490, 544 for producing average 
dark values. In other embodiments, there are more than two dark rows which are 
averaged. The fixed pattem noise reduction system of the present invention is not 
limited to two dark rows nor to any other particular number. The system 540 in Figure 

30 22 can be modified to average any number of dark rows prior to the subtraction from 

the exposed pixel by the summing amplifier 496. 
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Figure 23 illustrates one embodiment of an on-the-fly fixed pattern noise 
reduction process utilizing multiple dark rows 490, 544. The process of Figure 23 may 
be performed by the system 540 of Figure 22. In Figure 23, in a process block 550, the 
system 540 reads three pixels in a current column: a dark pixel from each of the two 
5 dark rows 490, 544 and an exposed pixel in an exposed row 186. In a process block 
552, the first summing amplifier 552 averages the values associated with the two dark 
pixels. In a process block 554, the second summing amplifier 496 subtracts the 
averaged dark pixel value from a value associated with the exposed pixel. In a process 
block 546, the system 540 clips the difference found in block 554 to zero if the 
10 difference is negative. In a process block 558, the system 540 advances to the next 

column. 

In a decision block 560, the system 540 determines whether the column shift 
register 146 has reached the last pixel in a row 186. If the column shift register 146 has 
not reached the last pixel in a row 186, then the system 540 returns to process block 550 

15 to process the next column. If the column shift register 146 has reached the last pixel in 
a row 186, then the system 540 advances to the next row 1 86* in a process block 562. In 
a decision block 564, the system 540 determines whether the row shift register 180 has 
reached the last row in a frame of pixel sensor elements. If the row shift register 180 
has not reached the last row in a frame of sensor elements, then the system 540 returns 

20 to process block 550 to process the first colxmm in the next row 105^ If the row shift 
register 180 has reached the last row in the frame, then the system 540 may wait for a 
fiuther command in an end block 566. 
Bad Pixel Correction 

In one embodiment, the imaging system contains a signal processing circuit 624 

25 that allows bad pixel correction as shown in Figure 24. Figure 24 demonstrates an 

example of a CMOS integrated circuit imaging device implementation that 
accommodates on-the-fly bad pixel correction. The illustration presents an 
implementation for a monochrome on-the-fly bad pixel correction when windowing and 
sub-sampling are not active. However, the present invention is not Umited to a 

30 monochrome imager, nor is limited in its mode of operation and can be realized to 

support window and sub-samphng. 
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In the embodiment shown in Figure 24, a monochrome imaging system 600 
employs a CMOS integrated circuit. The bad pixel correction system may be appUed to 
CCD integrated circuits and/or different color filters. 

Figure 24 illustrates one embodiment of a CMOS integrated circuit imaging 
5 system 600 that accommodates on-the-fly bad pixel correction. In Figure 24, the 

system 600 comprises a row shift register 180, a column shift register 146, buffers 188, 
188', 152, row control lines 186, 186', colunm control lines 602, 602', 602", a delay 
604, a multiplexer 606, an amphfier 608, a video out line 610, an external memory 
control 612, a row comparator 616, a column comparator 614, an AND gate 622, a row 

10 comparator 618, a column counter 620, a row control line 630 and a column control line 
632. Other circuit configurations may be implemented in accordance with the present 
invention. In another embodiment of Figure 24, a pipeline delay of two or more pixels 
is utiUzed. In a pipeline approach described below, a delay is not needed because one 
or more lines are stored such that the previous pixels are readily available for bad pixel 

15 correction. 

In Figure 24, the row control line 630 inputs the total number of rows into the 
row counter 618, and the column control line 620 inputs the total number of columns 
into the column counter 620. The color imaging system 600 provides an on-the-fly bad 
pixel correction process that is performed during the analog sensor readout stage. The 

20 signal processing circuit 624 identifies the addresses of the bad pixels via a hst stored in 

an extemal memory accessed by the extemal memory control 612 and corrects any bad 
pixels on-the-fly. The list may be stored in a memory either on-chip or off-chip. 

Figure 25 illustrates a flowchart for the on-the-fly bad pixel correction of Figure 
24. The detection of bad pixels is performed off-line and a bad pixel list is stored in an 

25 extemal memory which is accessed by the extemal memory control 612. The 
correction process utiUzes the Hst and replaces a defect pixel with its adjacent neighbor. 
In a block 630 of Figure 25, the circuit 600 reads consecutive horizontal pixels 
and amplifies the pixel readouts with an amphfier 608. The delay 604 provides a 
temporary pixel value storage. In a block 632, the row address comparator 616 and the 

30 column address comparator 614 compare the readout pixel address firom the row and 

column counters 618, 620 with the stored list of defective pixel addresses. In a decision 
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block 634, the circuit 624 determines if the current pixel is a bad one. If the 
comparison is positive, the output of the AND gate is 1, and the current pixel is 
replaced by the previous adjacent pixel that is stored in the delay 604, as shown in a 
block 636. If the comparison is negative, the output of the AND gate is 0, and the 
colunrn shift register 146 advances to the next column in a block 638. The blocks 640- 
646 are substantially similar to the blocks 214-220 in Figure 6 described above. 

The on-the-fly bad pixel correction embodiment may be combined with various 
readout modes such as windowing and sub-sampling, and additional signal processing 
such as on-the-fly color interpolation. The bad pixel correction system of the present 
invention is not limited to monochrome imaging systems and is appHcable to integrated 
circuit color imaging systems as well. The color imaging system may utihze any color 
filter scheme such as the primary color system (RGB) in a Bayer pattern, a 
complementary color system or others. The imaging system may also use a non- 
rectangle matrix of pixels. A similar implementation can provide on-the-fly bad pixel 
correction for color imaging systems and/or different pixel topography via modification 
to the readout control, the pixel neighborhood size and configuration. For example, a 
color imaging system with an RGB Bayer pattern may use 3 horizontally consecutive 
pixels, i.e., pipeline delay of 3 pixels and 2 registers for temporary storage. So each 
color component when defective, can be replaced by the corresponding pixel with the 
same color component. 

In other embodiments of the systems illustrated in Figures 4, 8, 13,15, 19, 20, 
22, the systems may further include one or more rows and/or columns of capacitors to 
read and store pixel sensor values to accommodate serial readout of pixels. Using 
capacitors to store values reduces the number and complexity of the lines (buses) 
coupled the pixel sensor elements. The size of the pixel sensor elements, the size of the 
sensor array, the space on the chip and the placement of the components are design 
choices of the manufacturer. 

The present invention does not require the integrated color imaging system to 
employ a simultaneous readout of the n-by-n pixel block. A ^pipeline' approach may be 
utiUzed instead of a parallel readout. The pipeline approach uses one or more analog 
line storage units, e.g., capacitors. For example, in one embodiment using the Bayer 
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color pattern, two line storage units are used. The first of two consecutive lines that is 
readout from the array is stored in the first line storage unit. The second Une is 
averaged with the stored Une to produce the RGB triplets, while a "first" line of the next 
two consecutive lines is readout and stored in the second line storage unit and so on. 
5 Thus, the two Une storage units are used in a 'ping pong' fashion. 

The invention may be embodied in other specific forms without departing fi*om 
its spirit or essential characteristics. The described embodiment is to be considered in 
all respects only as illustrative and not restrictive and the scope of the invention is, 
therefore, indicated by the appended claims rather than the foregoing description. All 
10 changes which come within the meaning and range of equivalency of the claims are to 

be embraced within their scope. 
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WHAT IS CLAIMED: 



1. A color imaging system providing on-the-fly color interpolation using 
analog signals to reconstruct colors during sensor readout, the imaging system 

5 comprising: 

an array of pixel sensor elements; 

a color filter including a plurality of color filter components organized in 
a predefined pattern, the color filter overlaying at least a portion of the array; 

a readout control circuit coupled to the array; and 
10 an array controller coupled to the array, wherein the readout circuit and 

the array controller reconstruct color components for at least a portion of the 
array while the readout control circuit is reading at least said portion of the array. 

2. The system of Claim 1 , fiirther comprising: 

a comparator circuit adapted to compare an address of a pixel sensor 
15 element currently being read by the readout control circuit with a stored hst of 

defective pixel sensor addresses; and 

at least one delay element for storing at least one previous analog pixel 
value read by the readout control circuit, wherein if the address of the current 
pixel sensor element matches a defective pixel address in the stored hst, the 
20 readout control circuit reads the previous analog pixel value. 

3. The system of Claim 1, wherein the readout control circuit is adapted to 
read a plurality of pixel sensor elements in parallel. 

4. The system of Claim 1, fixrther comprising: 

a first analog line storage unit, the first analog line storage imit being 
25 adapted to store a first hne readout from the array; and 

a second analog line storage unit, the second analog line storage unit 
being adapted to store a third Hne readout fi*om the array, wherein the readout 
control circuit averages a second consecutive line readout fi-om the array with 
the first line readout stored in the first analog line storage unit to produce a first 
30 red-green-blue (RGB) triplet, the readout control circuit averaging a fourth 
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consecutive line readout from the array with the third hne readout stored in the 
second analog line storage unit to produce a second RGB triplet. 

5 . The system of Claim 4, wherein the analog storage units are capacitors. 

6. The system of Claim 1, wherein the readout control circuit is 
programmable to read a first pixel element in a first mode and to read a second pixel 
element in a second mode. 

7. The system of Claim 1, wherein the pixel sensor elements form a portion 

of a charge coupled device. 

8. The system of Claim 1, wherein the pixel sensor elements form a portion 
of a complementary metal oxide semiconductor device. 

9. The system of Claim 1, fiirther comprising a first programmable gain 
ampUfier adapted to amphfy a first color readout signal a first amount and a second 
programmable gain amphfier adapted to amplify a second color readout signal a second 
amount. 

10. The system of Claim 9, wherein the programmable gain amplifiers are 
implemented as a separate stage. 

11. The system of Claim 9, wherein the progranmiable gain ampUfiers are 
contained within a pixel circuitry of the array. 

12. The system of Claim 9, wherein the programmable gain amphfiers are 
within a plurality of column buffers. 

13. The system of Claim 9, wherein the programmable gain ampUfiers have 

different transfer fimctions. 

14. The system of Claim 1, wherein at least a portion of the pixel sensor 

elements are active. 

15. The system of Claim 1, wherein at least a portion of the pixel sensor 

elements are passive. 

16. The system of Claim 1, wherein at least a first pixel sensor element is 
associated with a different color filter component than a neighboring pixel sensor 
element. 

17. The system of Claim 1, wherein the predefined pattern is a Bayer color 
configuration. 
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18. The system of Claim 1, wherein the predefined pattem comprises the 
colors of red, blue and green. 

19. The system of Claim 1, wherein the predefined pattem comprises the 
colors of yellow, cyan and magenta. 

20. The system of Claim 1 , further comprising a micro-lenses layer. 

21. The system of Claim 1, wherein the readout control circuit and the array 
controller process a first set of pixel sensor elements and then process a second set of 
pixel sensor elements, such that the second set of pixel sensor elements overlaps a 
portion of the first set of pixel sensor elements. 

22. The system of Claim 1, wherein the readout control circuit and the array 
controller process a first set of pixel sensor elements and then process a second set of 
pixel sensor elements, such that the second set of pixel sensor elements does not overlap 
the first set of pixel sensor elements. 

23. The system of Claim 1 , wherein the readout control circuit and the array 
controller process a first set of pixel sensor elements, skip a second set of pixel sensor 
elements and process a third set of pixel sensor elements. 

24. The system of Claim 1, wherein the readout control circuit and the array 
controller only processes a sub-region of the array of pixel sensor elements. 

25. The system of Claim 1, fiirther comprising a television coupled to said 

readout control circuit. 

26. The system of Claim 1, further comprising a personal computer coupled 

to said readout control circuit. 

27. The system of Claim 1, further comprising a monitor coupled to said 

readout control circuit. 

28. The system of Claim 1, further comprising a camera coupled to said 

readout control circuit. 

29. A method of interpolating color components of an array of pixel sensor 
elements, said method comprising: 

reading a portion of an array of pixel sensor elements; and 
reconstructing color components for at least a portion of the array while 
said portion of the array is being read. 
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30. The method of Claim 29, further comprising: 

comparing an address of a pixel sensor element currently being read by a 
readout control circuit with a stored list of defective pixel sensor addresses; and 

storing at least one previous analog pixel value read by the readout 
control circuit, wherein if the address of the current pixel sensor element 
matches a defective pixel address in the stored list, the readout control circuit 
reads the previous analog pixel value. 

31. The method of Claim 29, wherein reconstructing color components is 

performed in real-time. 

32. The method of Claim 29, wherein reconstructing color components is 

performed in an analog domain. 

33. The method of Claim 29, wherein the act of reading includes reading a 
first set of pixel sensor elements and then reading a second set of pixel sensor elements, 
such that the second set of pixel sensor elements overlaps a portion of the first set of 
pixel sensor elements. 

34. The method of Claim 29, wherein the act of reading includes reading a 
first set of pixel sensor elements and then reading a second set of pixel sensor elements, 
such that the second set of pixel sensor elements does not overlap a portion of the first 
set of pixel sensor elements. 

35. The method of Claim 29, wherein the act of reading includes reading a 
first set of pixel sensor elements, skipping a second set of pixel sensor elements and 
reading a third set of pixel sensor elements. 

36. The method of Claim 34, further comprising: 

summing a plurality of values associated with a plurality of pixel sensor 
elements associated Avith a first color to produce a first color component; and 

summing a plurality of values associated with a plurality of pixel sensor 
elements associated with a second color to produce a second color component. 

37. The method of Claim 29, wherein the act of reading includes reading 
only a sub-region of the array of pixel sensor elements. 

38. A color imager comprising: 
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a first light sensor which generates a first analog output signal related to 
the amount of a first color of Ught sensed; 

a second Ught sensor which generates a second analog output signal 
related to the amount of said first color of Ught sensed; and 

an interpolation circuit configured to receive said first output signal and 
said second output signal, wherein said interpolation circuit provides an 
interpolation signal on the fly based on at least said first analog output signal 
and said second analog output signal. 

39. A method of interpolating a color value in the analog domain in real- 
time, comprising: 

receiving a first analog signal corresponding to the output of a first pixel 
element in an imager, the first pixel element used to sense light intensity of a 
first color; 

receiving a second analog signal corresponding to the output of a second 
pixel element in the imager, the second element spaced from the first pixel 
element, wherein the second pixel element is used to sense Ught intensity of the 
first color; and 

generating an analog interpolation signal, the analog interpolation signal 
used to recreate a color value in real-time for a location situated between the first 
and second pixel elements based on the first analog signal and the second analog 
signal. 

40. The method as defined in Claim 39, further comprising generating an 
image based on at least the first analog signal, the second analog signal, and the analog 
interpolation signal. 

41 . The method as defined in Claim 39, further comprising: 
reading a third pixel element located in a line of pixel elements; 
skipping a fourth pixel element located in the line of pixel elements; and 
reading a fifth pixel element located in the line of pixel elements. 

42. The method as defined in Claim 39, fiirther comprising performing a 
windowing operation by reading only a subset of pixel elements during a read operation 
of the imager. 
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METHOD AND APPARATUS FOR COLOR INTERPOLATION 

Abstract of the Disclosure 

An imaging method and system that flexibly accesses light sensor elements and 
5 processes imaging signals. The imaging system comprises an array of pixel sensor 

cells, an array controller and a readout control circuit. The imaging system provides 
color interpolation. 
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USE THE GREEN ODD OUTPUT PATH 
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SECOND RED GAIN AMPLIFIER 

AMPLIFY VALUE VIA 
THIRD RED GAIN AMPLIFIER 



SELECT THE GREEN PIXEL THAT 

RESIDES IN CURRENT ROW 
OMIT THE GREEN PIXEL THAT 
RESIDES IN THE FOLLOWING ROW 




426 




42S 



-4J2 





440 



AMPLIFY VALUE VIA 
FIRST BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
SECOND BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
THIRD BLUE GAIN AMPLIFIER 




447 




444 



'452 



ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 




ADVANCE TO NEXT ROW: 
ROW-ROW+1 



EXCEEDING LAST ROW 
^WITH RED IN A FRAME ^ 

9 



,N0 



76A 



1 



4J4- 



4J6- 



AMPLIFY VALUE ViA 
FIRST GREEN GAIN AMPLiRER 

AMPLIFY VALUE VIA 
SECOND GREEN GAIN AMPLIFIE 

AMPLIFY VALUE VIA 
THIRD GREEN GAIN AMPLIFIER. 





450^ 



44S^ 



'446 



AMPLIFIED RED-f 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



F/a 76B 



o 
a 

m 
m 
a 

H 

ru 
O 

ry 
□ 
i;3 



READ 2x2 PIXEL BLOCK: 
RG or GR or GB or BG 
GB BG RG GR 



42 J - 



422 




RED PIXEL 



4JO 






AMPLIFY VALUE VIA 
FIRST RED GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
SECOND RED GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
THIRD RED GAIN AMPLIFIER 



AVERAGE TWO GREEN PIXELS 
VIA SUMMING AMPLIFIER: 
(G1 + G2)/2 



AMPLIFY VALUE VIA 
FIRST BLUE GAIN AMPLIFIER 




424 



-426 



42a 



4^2 



AMPLIFY VALUE VIA 
SECOND BLUE GAIN AMPLIFIER 



AMPLIFY VALUE VIA 
THIRD BLUE GAIN AMPLIFIER 




440 



-447 



444 



'432 



ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 




ADVANCE TO NEXT ROW; 
R0W=R0W+1 



'458 



EXCEEDING LAST ROW 
JVITH RED IN A FRAME 

7 



..NO 



YES 



.STOP 




^460 




AMPLIFY VALUE VIA 
FIRST GREEN GAIN AMPLIFIER 

AMPLIFY VALUE VIA 
SECOND GREEN GAIN AMPLIFIE 

AMPLIFY VALUE VIA 
THIRD GREEN GAIN AMPLIFIER 






448^ 



-446 



AMPLIFIED RED+ 
AMPLIFIED GREEN + 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
AMPLIFIED GREEN+ 
AMPLIFIED BLUE 



AMPLIFIED RED+ 
AMPLIFIED GREEN+ 
AMPLIFIED BLUE 




~500 



-U2 



'746 



LINE SYNC 



^^t PIXEL CLOCK 



DARK 

ROW ^-^^N Vn!— I 

READOUT^^-^' 
CONTROL 

W4 , 



COLUMN READOUT 
SHIFT REGISTER 



SUBTRACTION 
BY SUMMING 

VIDEO OUT 




r/a 20 



READ 2 PIXELS IN CURRENT 
COLUMN, THE DARK PIXEL AND 
THE EXPOSED PIXEL; 
D or D or D 
RGB 



1 



-322 



SUBTRACT DARK VALUE FROM 
EXPOSED PIXEL VALUE: 
R-D or G-D or B-D 



-324 



CLIP TO ZERO IF NEGATIVES 



I 



-326 



ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 




328 



3JO 



ADVANCE TO NEXT ROW; 
R0W=R0W+1 



-3J2 



EXCEEDING LAST ROW ING 
IN A FRAME 

9 



F/a 27 



f42\ 
LINE SYNC 



PIXEL CLOCK 



> 



-MS 



COLUMN READOUT 
SHIFT REGISTER 



VIDEO OUT 




F/a 22 



READ 3 PIXELS IN CURRENT 
COLUMN, 2 DARK PIXELS AND 
THE EXPOSED PIXEL 
D1 D1 D1 
D2 D2 D2 
RGB 



I 



AVERAGE TWO DARK PIXELS: 
AD=(DH-D2)/2 



-552 



I 



-554 



SUBTRACT AVERAGE DARK VALUE 
FROM EXPOSED PIXEL VALUE: 
R-AD or G-AD or B-AD 



CLIP TO ZERO IF NEGATIVE 




■555 



555 



ADVANCE TO NEXT COLUMN: 
C0LUMN=C0LUMN+1 



550 




562 



ADVANCE TO NEXT ROW: 
ROW=ROW+ 1 




F/G, 2 J 
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ADVANCE TO NEXT COLUMN; 
COLUMN = COLUMN + 1 




ADVANCE TO NEXT ROW: 
ROW = ROW+1 
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Attorney's Docket No. YMEDIA.OOl A 



DECLARATION - USA PATENT APPLICATION 



As a below named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below next to my name; 

I believe I am the original, first and sole inventor of the subject matter which is claimed 
and for which a patent is sought on the invention entitled METHOD AND APPARATUS FOR 
COLOR INTERPOLATION; the specification of which is attached hereto; 

I hereby state that I have reviewed and understand the contents of the above identified 
specification, including the claims, as amended by any amendment referred to above; 

I acknowledge the duty to disclose information which is material to patentability as 
defined in Title 37, Code of Federal Regulations, § L56; 

I hereby claim the benefit under Title 35, United States Codes § 119(e) of any United 
States provisional application(s) hsted below. 

Application No.: 60/149,796 Filing Date: August 19, 1999 

I hereby declare that all statements made herein of my own knowledge are true and that 
all statements made on information and behef are beUeved to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willful, false statements may jeopardize the vaUdity of the application or any 
patent issued thereon. 



Full name of sole inventor: Sarit Neter 



Residence: 15 Highland View, Irvine, C A 92612 

Citizenship: Israeli 

Post Office Address: same as above 



Send Correspondence To: 

KNOBBE, MARTENS, OLSON & BEAR, LLP 

Customer No. 20,995 

H:\DOCS\ACC\ACC-1470.DOC:smn 
012100 



Inventor's signature 




Date 



0^0 



YMEDIA.001A 



PATENT 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 

Applicant : Sarit Neter ) 

App. No. : Unknown ) 

Filed : Herewith ) 

For : METHOD AND APPARATUS FOR ) 

COLOR INTERPOLATION ) 

Examiner : Unknown ) 

ESTABLISHMENT OF RIGHT OF ASSIGNEE TO TAKE ACTION 

AND 

REVOCATION AND POWER OF ATTORNEY 

Assistant Commissioner for Patents 
Washington, D.C. 20231 

Dear Sir: 

The undersigned is empowered to act on behalf of the assignee below (the "Assignee"). 
A true copy of the original Assignment of the above-captioned application from the inventor(s) 
to the Assignee is attached hereto. This Assignment represents the entire chain of title of this 
invention from the Inventor to the Assignee. 

I declare that all statements made herein are true, and that all statements made upon 
information and beUef are beUeved to be true, and further, that these statements were made with 
the knowledge that willful, false statements and the like so made are punishable by fine or 
imprisonment, or both, under 18 U.S.C. § 1001, and that willful, false statements may jeopardize 
the validity of the application, or any patent issuing thereon. 

The undersigned hereby revokes any previous powers of attorney in the subject 
apphcation, and hereby appoints the registrants of Knobbe, Martens, Olson & Bear, LLP, 620 
Newport Center Drive, Sixteenth Floor, Newport Beach, California 92660, Telephone 
(949) 760-0404, Customer No. 20,995, as its attorneys with full power of substitution and 



App. No. : Unknown 
Filed : Herewith 

revocation to prosecute this appHcation and to transact all business in the U.S. Patent and 
Trademark Office connected herewith. This appointment is to be to the exclusion of the inventor 
and his attomeys in accordance with the provisions of 37 C.F.R. § 3.71. 
Please use Customer No. 20,995 for all communications. 



Y MEDIA COIIPQRATION 

Dated: 

Ian Olsen 
Title: President & CEO 




Address: 23 1 72 Plaza Pointe Drive, Suite 285 
Laguna Hills, CA 92653 
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